Establishment of audio-based network sessions with non-registered resources

ABSTRACT

The present disclosure is generally directed to increasing the scalability of onboarding network resources, such as a digital component, to a voice-based network. The system enables the navigating of and interaction with digital components using voice or speech input and output interfaces on a computing device. The system can receive and processes an input audio signal to identify a digital component. The system enables voice-based interaction with the previously unregistered digital component via the input and output interfaces.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a U.S. National Stage filing under 37 USC §371 of P.C.T. Application No. PCT/US2017/037284, filed Jun. 13, 2017,and claims the benefit of priority thereto. The foregoing application ishereby incorporated by reference in its entirety.

BACKGROUND

A computer system can interact with network resources that areconfigured to interact with the computer system. To interact with thenetwork resource an interface can be designed for each network resource.The expansion of certain computer systems can be limited becausecustomized interfaces for each of the network resources cannot beefficiently created, and can be duplicative and wasteful of computingresources such as bandwidth, time, or electrical power. Accordingly, theexpansion of certain computer systems is not scalable using currentonboarding techniques for network resources.

SUMMARY

At least one aspect is directed to a system to retrieve digitalcomponents in a voice activated data packet based computer networkenvironment includes a natural language processor component. The naturallanguage processor component can be executed by a data processingsystem. The natural language processing component can receive, via aninterface of the data processing system, data packets that can includean input audio signal detected by a sensor of a computing device. Thenatural language processor component can parse the input audio signal toidentify a request, a content provider, and a trigger keywordcorresponding to the request. The system can include a direct actionapplication programming interface that can generate, based on thetrigger keyword, an action data structure. The action data structure canbe generated responsive to the request and the identified contentprovider. The system can include a navigation component. The navigationcomponent can establish a session with the identified content provider.The navigation component can render a digital component received, viathe session, from the content provider. The navigation component canselect an interaction model associated with the content provider. Thenavigation component can generate a data array based on the interactionmodel and the action data structure. The system can transmit, via theinterface of the data processing system, the data array to the remotedata processing system.

At least one aspect is directed to a method to retrieve and interactwith digital components in a voice activated data packet based computernetwork environment. The method can include receiving, by a naturallanguage processor component executed by a data processing system andvia an interface of the data processing system, data packets that caninclude an input audio signal detected by a sensor of a computingdevice. The method can include parsing, by the natural languageprocessor component, the input audio signal to identify a request, acontent provider, and a trigger keyword corresponding to the request.The method can include generating, by a direct action applicationprogramming interface, based on the trigger keyword, an action datastructure. The action data structure can be generated responsive to therequest and the content provider. The method can include establishing,by a navigation component, a session with the content provider. Themethod can include rendering, by the navigation component, a digitalcomponent received, via the session, from the remote data processingsystem. The method can include selecting, by the navigation component,an interaction model associated with the remote data processing system.The method can include generating, by the navigation component, a dataarray based on the interaction model and the action data structure. Themethod can include transmitting, by the data processing system and viathe interface of the data processing system, the data array to theremote data processing system.

The foregoing general description and following description of thedrawings and detailed description are exemplary and explanatory and areintended to provide further explanation of the invention as claimed.Other objects, advantages, and novel features will be readily apparentto those skilled in the art from the following brief description of thedrawings and detailed description.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings are not intended to be drawn to scale. Likereference numbers and designations in the various drawings indicate likeelements. For purposes of clarity, not every component may be labeled inevery drawing. In the drawings:

FIG. 1 illustrates a block diagram of an example system to select andinteract with digital via a computer network.

FIG. 2 illustrates a block diagram of an example method to retrievedigital components in the system illustrated in FIG. 1.

FIG. 3 illustrates a block diagram of an example data flow through thesystem illustrated in FIG. 1.

FIG. 4 illustrates a block diagram of an example computer system for usein the system illustrated in FIG. 1.

DETAILED DESCRIPTION

Following below are more detailed descriptions of various conceptsrelated to, and implementations of, methods, apparatuses, and systems ofretrieving and interacting digital components in an audio-based computernetwork. The various concepts introduced above and discussed in greaterdetail below may be implemented in any of numerous ways, as thedescribed concepts are not limited to any particular manner ofimplementation

The present disclosure is generally directed systems and methods thatincrease the scalability of onboarding network resources, such as adigital component, to a voice or image based network. The system caninclude a data processing system that enables the navigating of andinteraction with digital components, such as a webpage, portion thereof,or other online document using voice, image, or speech input and outputinterfaces on computing device. The system can receive and processes aspeech input (also referred to herein as an input audio signal) toidentify a digital component. The system can identify a content providerthat provides the digital component. The voice input (or other non-textinput such as an image input) can include verbal instructions associatedwith at least one resource of a defined type that can be accessed in thedigital component. The resource can be, include, or correspond to anaction to complete via the identified digital component or a specificitem or webpage contained in or otherwise identified by the digitalcomponent. The system can create a session with the content providerhosting the digital component. The system can render the digitalcomponent received from the content provider. The system can headlesslyrender the digital component without a user interface. The system cananalyze, using an interaction model, the rendered digital component tonavigate to the resource using the interaction model. The system canselect the interaction model from a first model type and a second modeltype. The first model type can be or include a generic model that canincorporate a first training data set that is based on a collection ofwebsites having resources of the same type as the resource identified inthe speech input. The second model type can be or include a specificmodel that incorporates a second training data set specific to resourcesof the digital component. Both of the model sets can include data fordetermining access to respective resources or navigating the digitalcomponent. The system can provide information related to the associatedresources to the computing device for one or more subsequent operationsbased on the received instructions to access the respective resource.The system can also update the interaction models based on one or moredeterminations made when navigating the digital component to access theresource or additional resources.

The system can input data into the digital component. The data can beprovided by a user and stored in a secure storage wallet on thecomputing device or system. The data can be provided to the systemautomatically during the creation of the session with the digitalcomponent or when required during the created session for one or moresubsequent operations.

The system can generate the generic models by obtaining and storinginformation regarding one or more of terminology, layout, categories andhyperlinks that are common to accessing a respective type of resource ona plurality of different digital components. The system can generate thespecific model by obtaining and storing information regarding one ormore of terminology, layout, menus, categories, and hyperlinks ofresources respective to the digital component identified in the speechinput.

Onboarding of network resources into a voice-activated network such thatother network resources can interact with the network resources can betechnologically difficult because a unique interface may need to becreated for each resource. For example, the owners of the resource mayneed to create an application program interface (API) that enables theprovider of the voice-based computing device to interact with theresource. The provider of the voice-based computing device may also needto generate a program that enables the computing device to interact withthe resource via the provided API. The provider of the voice-basedcomputing device can provide the API, which the resource owner canincorporate into their digital component. This onboarding process can betime consuming, computationally inefficient and may require thecooperation of the two parties.

The onboarding process may be required to conduct a transaction with theresource, to gain access to a secure part of the resource, or exchangesensitive information. Techniques for interacting with these resourceswith a voice-input interface to conduct transactions and exchange datamay require tight integration of the resource interface of thevoice-based computing device so that a computing device can navigatethrough a resource using a speech command issued to the particular userinterface module. Such resource specific integration that may need to bepredefined between the interface module and the respective resource canbe necessary to ensure that the user of the computing device is providewith a secure, fast, and seamless experience. This is because priorsecurity configuration of how secure data is obtainable to the interfacemodule (e.g., from a secure wallet on the computing device), as well asprior integration with the particular resource may be required so thatthe interface module knows the layout of the resource inputs,components, or other interaction objects.

Accordingly, interfacing with resources that have not gone through theonboarding process to establish a tight integration between thevoice-based computing device and the resource may not be possible. Thecompletion of actions with a resource using the voice-based computingdevice may not be possible without an integration with each and everyresource.

Furthermore, the voice (or image or video) based computing device canuse a number of speech recognition based interface modules. Depending onthe module that a particular interface module uses, a pre-definedintegration with each respective interface module may be required foreach resource that the voice-based computing device could interact with.

Techniques that provide such integrations with voice-based computingdevice to enable navigation and interaction with a resource can sufferfrom the above explained drawbacks of requiring very specific and closepre-defined integration for each resource. These techniques may not bescalable and cannot be applied to all resources. For example, a websiteor a domain that offer the functionality to allow a speech controlleduser interface module to navigate the website, may need to limit thefunctionality to one type of interface module rather than being able tointeract with any voice input recognition module.

Accordingly, there exists a need for provision of a generic and scalabletechnique enabling voice-based computing devices to navigate andinteract with all resources using a speech, voice, or image inputrecognition interface module.

The present disclosure provides the following technical steps forenabling a real time and scalable technique operable with all resourcesthat provide products, services, or other actions. A user can initiate acommand via a voice-based computing device, such as “OK, I want toobtain product X on website Y, please let me know the price andavailability.” The system (e.g., a data processing system) can usenatural language processing to parse and interpret the input audiosignal. For example, the system can obtain a user's credentials forwebsite Y via a secure wallet. The system can initiate a session withwebsite Y. The server can headlessly render website Y. The system canthen navigate website Y using one of a specific or generic interactionmodel to obtain the price and availability to product X. The price andavailability can be provided to the user's computing device via anoutput audio file that is presented to the user via a transducer (e.g.,speaker) on or associated with the computing device. The user canconfirm the purchase or provide additional details to the system via asecond input audio signal that the system can parse. In this example,the system enabled the user to interact with website Y without requiringany specific integration between the system and the voice-basedcomputing device. The technique provides a method and system that can inreal time interact with a website, network resource, or other digitalcomponent using trained interaction models and without the specificintegration.

The techniques described herein can be generic and scalable to all typesof digital component and can enable the interaction of the dataprocessing system with the digital components without pre-integration orcoordination between the digital component provider and the provider ofthe voice-based computing device.

The above technical solution is provided by the technical features of aserver side headless rendering of respective digital components toestablish a mapping between the server and the domain of the website.Using as least one trained interaction model the system can identify thestructure, elements, input elements, and other components of the digitalcomponent. The above steps can occur seamlessly and automatically inreal time to further provide an effective end user experience ofprocessing the input audio signal to interact with a digital component.

FIG. 1 illustrates a block diagram of an example system 100 to selectand interact with digital components via a computer network. The system100 can include content selection infrastructure. The system 100 caninclude a data processing system 102. The data processing system 102 cancommunicate with one or more of a content provider computing device 106or client computing devices 104 via a network 105. The network 105 caninclude computer networks such as the Internet, local, wide, metro, orother area networks, intranets, satellite networks, and othercommunication networks such as voice or data mobile telephone networksand combinations thereof. The network 105 can access informationresources such as web pages, web sites, domain names, or uniformresource locators that can be presented, output, rendered, or displayedon at least one computing device 104, such as a laptop, desktop, tablet,personal digital assistant, smart phone, home assistant device, portablecomputer(s), or speaker. For example, via the network 105 a user of thecomputing device 104 can access information or data provided by thecontent provider device 106.

The data processing system 102 can include an interface 110, a naturallanguage processor component 112, and a session handler component 114.The data processing system 102 can also include a direction actionapplication programming interface 116, a navigation component 118, andan audio signal generator component 122. The data processing system 102can also include a data repository 124 on which parameters 126, policies128, interaction models 130, and templates 132 are stored.

The network 105 can be used by the data processing system 102 to accessinformation resources such as web pages, web sites, domain names, oruniform resource locators that can be presented, output, rendered, ordisplayed by the client computing device 104. The web pages, web sites,and the other digital content stored or otherwise provided by thecontent provider device 106 can be referred to as digital components orcontent items. Via the network 105 a user of the client computing device104 can access information or data (e.g., digital components such ascontent items) provided by the content provider computing device 106.

The digital component can be rendered via a display device of thecomputing device 104 or can be rendered on the data processing system102. Rendering can include displaying the digital component or othercontent item on a display device that may or may not be part of thecomputing device 104. In some implementations, the computing device 104does not include a display device for rendering the digital component.For example, the computing device 104 can only render digital componentsby playing the digital component via a speaker of the computing device104. The data processing system 102 can act as an intermediary andenable the computing device 104 to interact with the digital componentin an audio based fashion.

The network 105 may be any type or form of network and may include anyof the following: a point-to-point network, a broadcast network, a widearea network, a local area network, a telecommunications network, a datacommunication network, a computer network, an ATM (Asynchronous TransferMode) network, a SONET (Synchronous Optical Network) network, a SDH(Synchronous Digital Hierarchy) network, a wireless network and awireline network. The network 105 may include a wireless link, such asan infrared channel or satellite band. The topology of the network 105may include a bus, star, or ring network topology. The network mayinclude mobile telephone networks using any protocol or protocols usedto communicate among mobile devices, including advanced mobile phoneprotocol (“AMPS”), time division multiple access (“TDMA”), code-divisionmultiple access (“CDMA”), global system for mobile communication(“GSM”), general packet radio services (“GPRS”) or universal mobiletelecommunications system (“UMTS”). Different types of data may betransmitted via different protocols, or the same types of data may betransmitted via different protocols.

The system 100 can include at least one data processing system 102. Thedata processing system 102 can include at least one logic device such asa computing device having a processor to communicate via the network105, for example with the computing device 104 or the content providerdevice 106 (content provider 106). The data processing system 102 caninclude at least one computation resource, server, processor or memory.For example, the data processing system 102 can include a plurality ofcomputation resources or servers located in at least one data center.The data processing system 102 can include multiple, logically-groupedservers and facilitate distributed computing techniques. The logicalgroup of servers may be referred to as a data center, server farm or amachine farm. The servers can also be geographically dispersed. A datacenter or machine farm may be administered as a single entity, or themachine farm can include a plurality of machine farms. The serverswithin each machine farm can be heterogeneous—one or more of the serversor machines can operate according to one or more type of operatingsystem platform.

Servers in the machine farm can be stored in high-density rack systems,along with associated storage systems, and located in an enterprise datacenter. For example, consolidating the servers in this way may improvesystem manageability, data security, the physical security of thesystem, and system performance by locating servers and high performancestorage systems on localized high performance networks. Centralizationof all or some of the data processing system 102 components, includingservers and storage systems, and coupling them with advanced systemmanagement tools allows more efficient use of server resources, whichsaves power and processing requirements and reduces bandwidth usage.

The system 100 can include, access, or otherwise interact with at leastone content provider device 106. The content provider device 106 caninclude at least one logic device such as a computing device having aprocessor to communicate via the network 105. The content providerdevice 106 can include at least one computation resource, server,processor or memory. For example, the content provider device 106 caninclude a plurality of computation resources or servers located in atleast one data center.

The content provider computing device 106 can provide digital componentsto the data processing system 102 and the computing device 104. Thedigital components can be webpages that include graphics, text,hyperlinks, and machine executable instructions. The digital componentscan be displayed visually to an end user, such as, via a web browserrendering the webpage and displaying the rendered webpage to the user ona display. The digital component can be or include a webpage includingan offer, good, service, or information. For example, the digitalcomponent can be a website that sales clothing.

The computing device 104 can include, interface, or otherwisecommunicate with at least one sensor 134, transducer 136, audio driver138, or pre-processor 140. The sensor 134 can include, for example, anambient light sensor, proximity sensor, temperature sensor,accelerometer, gyroscope, motion detector, GPS sensor, location sensor,microphone, or touch sensor. The transducer 136 can include a speaker ora microphone. The audio driver 138 can provide a software interface tothe hardware transducer 136. The audio driver 138 can execute an audiofile or other instructions provided by the data processing system 102 tocontrol the transducer 136 to generate a corresponding acoustic wave orsound wave. The pre-processor 140 can be configured to detect a keywordand perform an action based on the keyword. The pre-processor 140 canfilter out one or more terms or modify the terms prior to transmittingthe terms to the data processing system 102 for further processing. Thepre-processor 140 can convert the analog audio signals detected by themicrophone into a digital audio signal, and transmit one or more datapackets carrying the digital audio signal to the data processing system102 via the network 105. In some cases, the pre-processor 140 cantransmit data packets (or other protocol based transmission) carryingsome or all of the input audio signal responsive to detecting aninstruction to perform such transmission, such as “OK,” “start,” oranother wake word. The instruction can include, for example, a triggerkeyword or other keyword or approval to transmit data packets comprisingthe input audio signal to the data processing system 102. In some cases,the primary user interface of the computing device 104 may be amicrophone and speaker.

Types of actions can include, for example, services, products,reservations, or tickets. Types of actions can further include types ofservices or products. For example, types of services can include carshare service, food delivery service, laundry service, maid service,repair services, or household services. Types of products can include,for example, clothes, shoes, toys, electronics, computers, books, orjewelry. Types of reservations can include, for example, dinnerreservations or hair salon appointments. Types of tickets can include,for example, movie tickets, sports venue tickets, or flight tickets. Insome cases, the types of services, products, reservations or tickets canbe categorized based on price, location, type of shipping, availability,or other attributes.

The client computing device 104 can be associated with an end user thatenters voice queries as audio input into the client computing device 104(via the sensor 134) and receives audio output in the form of a computergenerated voice that can be provided from the data processing system 102(or the content provider computing device 106) to the client computingdevice 104, output from the transducer 136 (e.g., a speaker). Thecomputer generated voice can include recordings from a real person orcomputer generated language. Rather than voice queries, the input canalso include one or more images or video segments generated or obtained(e.g., via the network 105) from the client computing device 104 andparsed by the data processing system 102 to obtain the same type ofinformation obtained by parsing of the voice queries. For example, theuser can take a picture of item they wish to purchase. The dataprocessing system 102 can perform machine vision on the image toidentify the contents of the image and generate a text string thatidentifies the contents of the image. The text string can be used as theinput query.

The data processing system 102 can include, interface, or otherwisecommunicate with at least one interface 110 that enables the dataprocessing system 102 to communicate with other devices, such as via thenetwork 105. The data processing system 102 can include, interface, orotherwise communicate with at least one natural language processorcomponent 112. The data processing system 102 can include, interface, orotherwise communicate with at least one direct action applicationprogramming interface (“API”) 116. The data processing system 102 caninclude, interface, or otherwise communicate with at least one sessionhandler 114. The data processing system 102 can include, interface, orotherwise communicate with at least navigation component 118. The dataprocessing system 102 can include, interface, or otherwise communicatewith at least one audio signal generator 122. The data processing system102 can include, interface, or otherwise communicate with at least onedata repository 124.

The data processing system 102 can include, interface, or otherwisecommunicate with a navigation component 118. The navigation component118 can enable a voice-based interaction between a computing device 104and a digital component, such as a website, provided by a contentprovider 106. The digital component provided by the content provider 106may not be configured to accommodate a voice-based interaction. Forexample, the digital component can be a webpage that includes text,images, video, input elements, and other non-audio elements.Additionally, there may be no previously established integration betweenthe digital component (or provider thereof) and the data processingsystem 102. The navigation component 118 can render the digitalcomponent, using for example, a headless browser or headless webkitrenderer, and identify input elements, text, and other data in thedigital component. When rendered with a headless renderer, the rendereddigital component does not require a graphical user interface tofunction. The navigation component 118 can interact with those elementsusing an interaction model. For example, the navigation component 118can input data arrays into the input fields, select and activate inputelements (e.g., navigation or submission buttons), and retrieve databased on the interaction model to complete an action as identified in aninput audio signal received from the computing device 104. As oneexample, for the input audio signal “buy two shirts,” the navigationcomponent 118 can generate a data array of “text=2.” The navigationcomponent 118 can also identify the input field and the “buy” button inthe headlessly rendered webpage. The navigation component 118 can enterthe text “2” into the input field and then select the “buy” button tocomplete the transaction.

The data repository 124 can include one or more local or distributeddatabases, and can include a database management system. The datarepository 124 can include computer data storage or memory and can storeone or more parameters 126, one or more policies 128, interaction models130, and templates 132 among other data. The parameters 126, policies128, and templates 132 can include information such as rules about avoice based session between the client computing device 104 and the dataprocessing system 102. The data repository 124 can also store contentdata that can include content items for audio output or associatedmetadata, as well as input audio messages that can be part of one ormore communication sessions with the client computing device 104. Theparameters 126 can include, for example, thresholds, distances, timeintervals, durations, scores, or weights.

The interaction models 130 can be generated and updated by thenavigation component 118. The data repository 124 can include multipleinteraction models. The interaction models 130 can be categorized intogeneric models and content provider specific models. The generic modelscan be further subdivided into different interaction or actioncategories. For example, the interaction models 130 can include ageneric model for different types of commercial website, such as ashopping website, a weather provider website, and a reservation bookingwebsite.

The interaction models 130 can also include specific models. Thespecific interaction models can be specific to a content provider 106 ora specific digital component provided by a specific content provider106. For example, for a specific website Y, the specific interactionmodel can know the placement of links, menus, how to navigate thewebsite, and how particular products and data is stored and classifiedwithin the website. The navigation component 118 can use thisinformation to navigate through the website and to provide data arraysto the website when interacting with the website to complete an actionor other transaction with the website.

The generic interaction models can be used when the navigation component118 has not had a predetermined number of interactions with the digitalcomponent or content provider 106 to generate a specific interactionmodel. The navigation component 118 can train the models (both thegeneric and the specific interaction models), by initially collectingdata from specific sessions (e.g., user sessions upon obtaining theirpermission or specific training sessions where a user interacts with thedigital component). For example, given an input audio signal, a user cancomplete the task of the input audio signal (e.g., “Ok, buy a shirt.”)The navigation component 118 can receive the inputs made by the user incompleting the action and build a model that enables the navigationcomponent 118 to identify the input elements, such as text fields andbuttons, that are used in completing the action. The specific models canbe trained on specific digital components and the generic models can betrained using a plurality of digital component within a given category.The training can enable the model to determine interaction data such asthe steps involved in purchasing a particular product, categories ofproducts, sorting of products, and how ordering is carried out on thedigital component. The models can enable the navigation component 118 tocorrectly identify and interact with the products or services that areidentified in the input audio signal.

Both types of interaction models 130 can be trained and updated duringand after sessions between the data processing system 102 and thecontent provider 106. For example, while using a generic model for adigital component, the navigation component 118 can build a specificmodel for the digital component. Once the specific interaction model forthe specific navigation component 118 is deemed to be reliable, forexample by being built with the data from a predetermined number ofsessions, the navigation component 118 can begin to use the specificinteraction model for the digital component rather than the genericmodel. The navigation component 118 can use the data from additional (ornew sessions) to updated the interaction models 130.

The interface 110, the natural language processor component 112, thesession handler 114, the direct action API 116, the navigation component118, or the audio signal generator component 122 can each include atleast one processing unit or other logic device such as programmablelogic array engine, or module configured to communicate with thedatabase repository or database 124. The interface 110, the naturallanguage processor component 112, the session handler 114, the directaction API 116, the navigation component 118, the audio signal generatorcomponent 122, and the data repository 124 can be separate components, asingle component, or part of the data processing system 102. The system100 and its components, such as a data processing system 102, caninclude hardware elements, such as one or more processors, logicdevices, or circuits.

The data processing system 102 can obtain anonymous computer networkactivity information associated with a plurality of computing devices104. A user of a computing device 104 can affirmatively authorize thedata processing system 102 to obtain network activity informationcorresponding to the user's computing device 104. For example, the dataprocessing system 102 can prompt the user of the computing device 104for consent to obtain one or more types of network activity information.The identity of the user of the computing device 104 can remainanonymous and the computing device 104 can be associated with a uniqueidentifier (e.g., a unique identifier for the user or the computingdevice 104 provided by the data processing system 102 or a user of thecomputing device). The data processing system 102 can associate eachobservation with a corresponding unique identifier.

The data processing system 102 can include an interface component 110designed, configured, constructed, or operational to receive andtransmit information using, for example, data packets. The interface 110can receive and transmit information using one or more protocols, suchas a network protocol. The interface 110 can include a hardwareinterface, software interface, wired interface, or wireless interface.The interface 110 can facilitate translating or formatting data from oneformat to another format. For example, the interface 110 can include anapplication programming interface that includes definitions forcommunicating between various components, such as software components.

The data processing system 102 can receive data packets or other signalthat includes or identifies an audio input signal. For example, the dataprocessing system 102 can execute or run the NLP component 112 toreceive or obtain the audio signal and parse the audio signal. Forexample, the NLP component 112 can provide for interactions between ahuman and a computer. The NLP component 112 can be configured withtechniques for understanding natural language and allowing the dataprocessing system 102 to derive meaning from human or natural languageinput. The NLP component 112 can include or be configured with techniquebased on machine learning, such as statistical machine learning. The NLPcomponent 112 can utilize decision trees, statistical models, orprobabilistic models to parse the input audio signal. The NLP component112 can perform, for example, functions such as named entity recognition(e.g., given a stream of text, determine which items in the text map toproper names, such as people or places, and what the type of each suchname is, such as person, location, or organization), natural languagegeneration (e.g., convert information from computer databases orsemantic intents into understandable human language), natural languageunderstanding (e.g., convert text into more formal representations suchas first-order logic structures that a computer module can manipulate),machine translation (e.g., automatically translate text from one humanlanguage to another), morphological segmentation (e.g., separating wordsinto individual morphemes and identify the class of the morphemes, whichcan be challenging based on the complexity of the morphology orstructure of the words of the language being considered), questionanswering (e.g., determining an answer to a human-language question,which can be specific or open-ended), semantic processing (e.g.,processing that can occur after identifying a word and encoding itsmeaning in order to relate the identified word to other words withsimilar meanings).

The NLP component 112 converts the audio input signal into recognizedtext by comparing the input signal against a stored, representative setof audio waveforms (e.g., in the data repository 124) and choosing theclosest matches. The set of audio waveforms can be stored in datarepository 124 or other database accessible to the data processingsystem 102. The representative waveforms are generated across a largeset of users, and then may be augmented with speech samples from theuser. After the audio signal is converted into recognized text, the NLPcomponent 112 matches the text to words that are associated, for examplevia training across users or through manual specification, with actionsthat the data processing system 102 can serve.

The audio input signal can be detected by the sensor 134 or transducer136 (e.g., a microphone) of the client computing device 104. Via thetransducer 136, the audio driver 138, or other components the clientcomputing device 104 can provide the audio input signal to the dataprocessing system 102 (e.g., via the network 105) where it can bereceived (e.g., by the interface 110) and provided to the NLP component112 or stored in the data repository 124.

The NLP component 112 can obtain the input audio signal. From the inputaudio signal, the NLP component 112 can identify at least one request orat least one trigger keyword corresponding to the request. The requestcan indicate intent or subject matter of the input audio signal. Thetrigger keyword can indicate a type of action likely to be taken. Forexample, the NLP component 112 can parse the input audio signal toidentify at least one request to leave home for the evening to attenddinner and a movie. The trigger keyword can include at least one word,phrase, root or partial word, or derivative indicating an action to betaken. For example, the trigger keyword “go” or “to go to” from theinput audio signal can indicate a need for transport. In this example,the input audio signal (or the identified request) does not directlyexpress an intent for transport, however the trigger keyword indicatesthat transport is an ancillary action to at least one other action thatis indicated by the request.

The NLP component 112 can parse the input audio signal to identify,determine, retrieve, or otherwise obtain the request and the triggerkeyword. For instance, the NLP component 112 can apply a semanticprocessing technique to the input audio signal to identify the triggerkeyword or the request. The NLP component 112 can apply the semanticprocessing technique to the input audio signal to identify a triggerphrase that includes one or more trigger keywords, such as a firsttrigger keyword and a second trigger keyword. For example, the inputaudio signal can include the sentence “I need someone to do my laundryand my dry cleaning.” The NLP component 112 can apply a semanticprocessing technique, or other natural language processing technique, tothe data packets comprising the sentence to identify trigger phrases “domy laundry” and “do my dry cleaning”. The NLP component 112 can furtheridentify multiple trigger keywords, such as laundry, and dry cleaning.For example, the NLP component 112 can determine that the trigger phraseincludes the trigger keyword and a second trigger keyword.

The NLP component 112 can parse the input audio signal to identify,determine, retrieve, or otherwise obtain an identification of a remotecontent provider 106 or remote data processing system 102 in a methodsimilar to the method the NLP component 112 uses to obtain the requestand trigger keyword. For example, the input audio signal including thephrase “ok, buy a red shirt from ABC” can be parsed to identify ABC as aseller of shirts. The data processing system 102 can then determine acontent provider 106 associated with ABC. The content provider 106 canbe a server that hosts ABC's website. The data processing system 102 canidentify a web address of ABC, such as “www.ABC.com.” The dataprocessing system 102 can transmit a confirmation audio signal to thecomputing device 104, such as “Are you referring to ABC of www.ABC.com?”Responsive to receiving a confirmation message from the computing device104, the data processing system 102 can initiate a session with thecontent provider 106 located at www.ABC.com.

The NLP component 112 can filter the input audio signal to identify thetrigger keyword. For example, the data packets carrying the input audiosignal can include “It would be great if I could get someone that couldhelp me go to the airport”, in which case the NLP component 112 canfilter out one or more terms as follows: “it”, “would”, “be”, “great”,“if”, “I”, “could”, “get”, “someone”, “that”, “could”, or “help”. Byfiltering out these terms, the NLP component 112 may more accurately andreliably identify the trigger keywords, such as “go to the airport” anddetermine that this is a request for a taxi or a ride sharing service.

The data processing system 102 can include a direct action API 116designed and constructed to generate, based on the trigger keyword, anaction data structure responsive to the request and the identifiedremote content provider 106. Processors of the data processing system102 can invoke the direct action API 116 to execute scripts thatgenerate a data structure to the content provider 106 request or order aservice or product, such as a car from a car share service. The directaction API 116 can obtain data from the data repository 124, as well asdata received with end user consent from the client computing device 104to determine location, time, user accounts, logistical or otherinformation to allow the data processing system 102 to perform anoperation, such as reserve a car from the car share service.

The direct action API 116 can execute a specified action to satisfy theend user's intention when the data processing system 102 interacts withthe digital component from the content provider device 106. Depending onthe action specified in its inputs, the direct action API 116 canexecute code or a dialog script that identifies the parameters requiredto fulfill a user request, which can be included into an action datastructure. Such code can look-up additional information or it canprovide audio output for rendering at the client computing device 104 toask the end user questions such as the user's preferred shirt size, tocontinue the above example where the input audio signal is “ok, by a redshirt.” The direct action API 116 can determine necessary parameters andcan package the information into an action data structure. For example,when the input audio signal is “ok, buy a red shirt,” that action datastructure can include the user's preferred shirt size.

The direct action API 116, upon identifying the type of request, canaccess the corresponding template from a template repository stored inthe data repository 124. The templates 132 can include fields in astructured data set that can be populated by the direct action API 116to further the operation of fulfilling or interacting with the digitalcomponent provided by the content provider 106. The direct action API116 can perform a lookup in the template repository to select thetemplate that matches one or more characteristic of the trigger keywordand request. For example, if the request corresponds to a request for acar or ride to a destination, the data processing system 102 can selecta car sharing service template. The car sharing service template caninclude one or more of the following fields: device identifier, pick uplocation, destination location, number of passengers, or type ofservice. The direct action API 116 can populate the fields with values.To populate the fields with values, the direct action API 116 can ping,poll or otherwise obtain information from one or more sensors 134 of thecomputing device 104 or a user interface of the computing device 104.For example, the direct action API 116 can detect the source locationusing a location sensor, such as a GPS sensor. The direct action API 116can obtain further information by submitting a survey, prompt, or queryto the end of user of the computing device 104. The direct action api116 can submit the survey, prompt, or query via interface 110 of thedata processing system 102 and a user interface of the computing device104 (e.g., audio interface, voice-based user interface, display, ortouch screen). Thus, the direct action API 116 can select a template forthe action data structure based on the trigger keyword or the request,populate one or more fields in the template with information detected byone or more sensors 134 or obtained via a user interface, and generate,create or otherwise construct the action data structure to facilitateperformance of an operation by the content provider 106.

The data processing system 102 can select the template based from thetemplate data structure based on various factors including, for example,one or more of the trigger keyword, request, type of content provider106, a category that the content provider 106 (e.g., taxi service,laundry service, flower service, retail service, or food delivery),location, or other sensor information.

To select the template based on the trigger keyword, the data processingsystem 102 (e.g., via direct action API 116) can perform a look-up orother query operation on the template database using the trigger keywordto identify a template data structure that maps or otherwise correspondsto the trigger keyword. For example, each template in the templatedatabase can be associated with one or more trigger keywords to indicatethat the template is configured to generate an action data structureresponsive to the trigger keyword that the data processing system 102can process to establish a communication session between the dataprocessing system 102 and the content provider 106.

To construct or generate the action data structure, the data processingsystem 102 can identify one or more fields in the selected template topopulate with values. The fields can be populated with numerical values,character strings, Unicode values, Boolean logic, binary values,hexadecimal values, identifiers, location coordinates, geographic areas,timestamps, or other values. The fields or the data structure itself canbe encrypted or masked to maintain data security.

Upon determining the fields in the template, the data processing system102 can identify the values for the fields to populate the fields of thetemplate to create the action data structure. The data processing system102 can obtain, retrieve, determine or otherwise identify the values forthe fields by performing a look-up or other query operation on the datarepository 124.

In some cases, the data processing system 102 can determine that theinformation or values for the fields are absent from the data repository124. The data processing system 102 can determine that the informationor values stored in the data repository 124 are out-of-date, stale, orotherwise not suitable for the purpose of constructing the action datastructure responsive to the trigger keyword and request identified bythe NLP component 112 (e.g., the location of the client computing device104 may be the old location and not be the current location; an accountcan be expired; the destination restaurant may have moved to a newlocation; physical activity information; or mode of transportation).

If the data processing system 102 determines that it does not currentlyhave access, in memory of the data processing system 102, to the valuesor information for the field of the template, the data processing system102 can acquire the values or information. The data processing system102 can acquire or obtain the information by querying or polling one ormore available sensors of the client computing device 104, prompting theend user of the client computing device 104 for the information, oraccessing an online web-based resource using an HTTP protocol. Forexample, the data processing system 102 can determine that it does nothave the current location of the client computing device 104, which maybe a needed field of the template. The data processing system 102 canquery the client computing device 104 for the location information. Thedata processing system 102 can request the client computing device 104to provide the location information using one or more location sensors134, such as a Global Positioning System sensor, WIFI triangulation,cell tower triangulation, Bluetooth beacons, IP address, or otherlocation sensing technique.

In some cases, the data processing system 102 can identify a remotecontent provider 106 with which to establish a session based on thetrigger keyword or request. To identify the content provider 106 basedon the trigger keyword, the data processing system 102 can perform alookup in the data repository 124 to identify a content provider 106that maps to the trigger keyword. For example, if the trigger keywordincludes “ride” or “to go to”, then the data processing system 102(e.g., via direct action API 116) can identify a content provider 106(or web address thereof) corresponding to Taxi Service Company A. Thedata processing system 102 can select the template from the templatedatabase based on the identified content provider 106. The dataprocessing system 102 can identify the content provider 106 byconducting an internet based search.

The data processing system 102 can include, execute, access, orotherwise communicate with a session handler component 114 to establisha communication session between with the computing device 104 and thedata processing system 102. The communication session can also refer toone or more data transmissions between the data processing system 102and the content provider 106. The communication session between thecomputing device 104 and the data processing system 102 can include thetransmission of the input audio signal that is detected by a sensor 134of the computing device 104, and the output signal transmitted by thedata processing system 102 to the computing device 104. The dataprocessing system 102 (e.g., via the session handler component 114) canestablish the communication session responsive to receiving the inputaudio signal. The data processing system 102 can set a duration for thecommunication session. The data processing system 102 can set a timer ora counter for the duration set for the communication session. Responsiveto expiration of the timer, the data processing system 102 can terminatethe communication session. The communication session between the dataprocessing system 102 and the content provider 106 can include thetransmission of the digital component from the content provider 106 tothe data processing system 102. The communication session between thedata processing system 102 and the content provider 106 can also includethe transmission of data arrays to the content provider 106. Thecommunication session can refer to a network-based communication sessionin which data (e.g., digital components, authenticating information,credentials, etc.) is transmitted between the data processing system 102and the content provider 106 and the data processing system 102 and thecomputing device 104.

The data processing system 102 can include, execute, or communicate withan audio signal generator component 122 to generate an output signal.The output signal can include one or more portions. The output signalcan include content identified from the digital component received fromthe content provider 106.

The audio signal generator component 122 can generate the output signalwith a first portion having sound corresponding to the first datastructure. For example, the audio signal generator component 122 cangenerate the first portion of the output signal based on one or morevalues populated into the fields of the action data structure by thedirect action API 116. In a taxi service example, the values for thefields can include, for example, 123 Main Street for pick-up location,1234 Main Street for destination location, 2 for number of passengers,and economy for the level of service.

The data processing system 102 (e.g., via interface 110 and network 105)can transmit data packets comprising the output signal generated by theaudio signal generator component 122. The output signal can cause theaudio driver component 138 of or executed by the computing device 104 todrive a speaker (e.g., transducer 136) of the computing device 104 togenerate an acoustic wave corresponding to the output

The content provider 106 can provide websites, goods, or services (allgenerally referred to as digital components) to the computing devices104 and the data processing system 102. The services and goods can beservices and goods that are provided physically (e.g., clothing, carservices, other consumables, etc.) and are associated with a digitalcomponent. For example, the digital component for a car service can be awebsite through which the user schedules the car service. The digitalcomponent associated with the service or goods can be digital componentsthat are used in the purchase, initiation, setup of the good orservices, or other transactions related to the goods and services.

The content provider 106 can include one or more keywords in the digitalcomponents. The keywords can be in meta tags, header strings, the bodyof the digital component, and links. After receiving the digitalcomponent, the navigation component 118 can analyze the keywords tocategorize the digital component (or content provider 106 associatedwith the digital component) into different categories. For example, thedigital component can be categorized into a news, retail, etc. categorythat identifies the general subject matter of the digital component. Thenavigation component 118 can select an interaction model from theinteraction models 130 at least partially based on the category of thedigital component.

The digital component can be rendered via a display device of thecomputing device 104 or can be rendered on the data processing system102. Rendering can include displaying the content item on a displaydevice. In some implementations, the computing device 104 does notinclude a display device for rendering the digital component. Forexample, the computing device 104 can only render digital components byplaying the digital component via a speaker of the computing device 104.The data processing system 102 can act as an intermediary and enable thecomputing device 104 to interact with the digital component in an audiobased fashion. The computing device 104 can include an application,script or program, installed at the client computing device 104, such asan app to communicate input audio signals to the interface 110 of thedata processing system 102. The application can also drive components ofthe computing device 104 to render output audio signals.

FIG. 2 illustrates a block diagram of an example method 200 to retrieveand interact with digital components in a voice activated, data packet(or other protocol) based computer network. FIG. 3 illustrates a blockdiagram of an example data flow through the system illustrated in FIG. 1during the course of the method 200 illustrated in FIG. 2. The method200 includes receiving an input audio signal (ACT 202). The method 200includes parsing the input audio signal to identify a request, a contentprovider, and a trigger keyword (ACT 204). The method 200 includesgenerating an action data structure (ACT 206). The method 200 includesestablishing a session with the content provider (ACT 208). The method200 includes rendering a received digital component (ACT 210). Themethod 200 includes selecting an interaction model (ACT 212). The method200 includes generating a data array based on the interaction model (ACT214). The method 200 includes transmitting the data array to the contentprovider (ACT 216).

As set forth above, and referring to FIGS. 2-3, the method 200 includesreceiving an input audio signal (ACT 202). The data processing system102 can receive the input audio signal 320 from the computing device104. The input audio signal 320 can be received by the data processingsystem 102 via a nlp component 112 over a network. The NLP can beexecuted by the data processing system 102. The data processing system102 can receive the input audio signal 320 as data packets that includethe input audio signal. The input audio signal can be detected by asensor, such as a microphone, of the computing device 104.

The method 200 includes parsing the input audio signal to identify arequest, a content provider, and a trigger keyword (ACT 204). The inputaudio signal can be parsed by the natural language processor component112. For example, the audio signal detected by the computing device 104can include “Okay device, I want a shirt from ABC Co.” In this inputaudio signal, the initial trigger keyword can include “okay device”,which can indicate to the computing device 104 to transmit an inputaudio signal to the data processing system 102. A pre-processor of thecomputing device 104 can filter out the terms “okay device” prior tosending the remaining audio signal to the data processing system 102. Insome cases, the computing device 104 can filter out additional terms orgenerate keywords to transmit to the data processing system 102 forfurther processing.

The data processing system 102 can identify a trigger keyword in theinput audio signal 320. The trigger keyword, which can be a phrase, caninclude, in the above example, “I want a shirt.” The trigger keyword canindicate a type of service or product (e.g., the shirt) and an action totake. The data processing system 102 can identify a request in the inputaudio signal. The request can be determined based on the terms “I want”.The trigger keyword and request can be determined using a semanticprocessing technique or other natural language processing technique. Thedata processing system 102 can identify the content provider 106 as ABCCo. The data processing system 102 can identify a website, IP address,or other network location of the content provider 106 associated withthe content provider 106, ABC Co.

The method 200 includes generating an action data structure (ACT 206).The direct action application programming interface can generate theaction data structure based on the trigger keyword. The action datastructure can also be generated responsive to the request and theidentified content provider 106. The action data structure can begenerated from or based on a template. The template can be selectedbased on the trigger keywords and the identified content provider 106.The generated action data structure can include information and datarelated to completing the action associated with the trigger keyword.For example, for “I want a shirt from ABC Co.” the template may indicatethe needed information related to the purchase of shirt can includesize, preferred color, preferred style, and preferred price range. Thedata processing system 102 can populate the fields in the action datastructure with values retrieved from the memory or based on responses bythe user to output signals transmitted from the data processing system102 to the computing device 104. The data processing system 102 canpopulate the secure fields such as user credentials from a secure walletthat can be stored on the data processing system 102 or the computingdevice 104. The data processing system 102 can request permission fromthe user to access the secure wallet before obtaining information fromthe secure wallet.

The method 200 includes establishing a session with the content provider(ACT 208). The data processing system 102 can establish a communicationsession 322 with the content provider 106 responsive to identifying thecontent provider 106 in the input audio signal. The communicationsession 322 can be established to receive a digital component from thecontent provider 106. The session can be established using the hypertexttransfer protocol. The session can be established with a request 324from the data processing system 102 to the content provider 106. Therequest 324 can be for a webpage transmitted in the response 325 to therequest 324.

The method 200 includes rendering a received digital component (ACT210). The received digital component can be rendered by the navigationcomponent 118 of the data processing system 102. FIG. 3 illustrates apartial rendering 300 of the digital component. Continuing the aboveexample, the digital component can be received in response to the inputaudio signal “I want a shirt from ABC Co.” The rendered digitalcomponent 300 can include input fields 302, buttons 304, menus, imagefields 306, images 308, and text 310 (generally referred to as thecomponents or elements of the digital component). Buttons, links, inputfields, and radio buttons can generally be referred to as inputelements. The digital component can be rendered without a graphical userinterface. For example, the digital component 300 can be an HTMLdocument that is rendered by a headless browser. The headless browser ofthe navigation component 118 can include a layout engine that can renderthe code of a digital component 300, such as the HTML and JavaScriptwithin the digital component. The navigation component 118, whenrendering the digital component in a headless fashion, can render thedigital component 300 to an image file that can be analyzed with amachine vision component of the navigation component 118.

The method 200 includes selecting an interaction model (ACT 212). Thenavigation component 118 can select an interaction model that isassociated with the content provider 106. The navigation component 118can select between two general types of interaction models. The firstmodel can be a generic model that may be the same for each contentprovider 106 associated with a particular category. For example, thedata processing system 102 may include a generic model for shoppingwebsites; a generic model for insurance websites; a generic model forhotel booking websites; and a generic model for food delivery websites.The second type of model can be specific to the content provider 106 (orthe digital component received from the content provider 106).

In addition, a second model that is a specific data model may also beused. For example, the model can be specific to ABC Co. Specific orspecial features such as placement of links to access products, specificmenus and how to navigate through them, how particular products arestored and classified within a website may be information that isinclude in the specific model. The navigation component 118 can use themodel to interpret the digital component 300. The navigation component118 can generate the specific interaction model after a predeterminednumber of sessions are established between the data processing system102 and the content provider 106. For example, initially the dataprocessing system 102 can use a generic model when interacting with agiven content provider 106. Data from the interactions can be used tobuild the specific interaction model. Once the data processing system102 has initiated a predetermined number of sessions and added thesession data to the specific model for the content provider 106, thedata processing system 102 can begin using the specific interactionmodel for the content provider 106. While the number of previouslyestablished sessions is below the predetermined number, the dataprocessing system 102 can continue to use the generic interaction modelwhen interacting with the content provider 106.

Using the selected model, the navigation component 118 can identify theinput fields 302, buttons 304, menus, image fields 306, images 308, andtext 310 of the digital component 300 by performing a machine visionanalysis of the saved image file of the digital component 300. Thenavigation component 118 can also identify the components of the digitalcomponent 300 by parsing the code of the digital component 300. Forexample, navigation component 118 can identify HTML tags within thedigital component 300. As one example, the navigation component 118 cansearch for the HTML tags <input> or <form> to identify the input field302.

When the navigation component 118 identifies an image or button, thenavigation component 118 can perform machine vision analysis on theimage or button to determine one or more characteristics of the image orbutton. The characteristics can include a determination of colors withinthe image (e.g., that the shirt illustrated in image 308 is a redshirt), identification of objects in the image (e.g., that image 308illustrates a shirt), or text or icons within the image or button (e.g.,that button 304 includes an arrow indicating “next” or if the button 304included the text “next”).

The method 200 includes generating a data array based on the interactionmodel (ACT 214). The data array can be generated by the navigationcomponent 118 based on information identified in the digital component300 using the interaction model. The data array can be generated usinginformation from the action data structure. For example, using theinteraction model, the navigation component 118 can determine the text310 states “size” and is associated with the input field 302. The actiondata structure can include an entry of “medium” in the field “size.” Thenavigation component 118 can include “medium” in the data array andinput the data array into the input field 302 to indicate that a shirtof medium size should be selected.

The method 200 includes transmitting the data array to the contentprovider (ACT 216). The data array 330 can be input into the input field302. The data array 330 can be transmitted to the content provider 106responsive to the navigation component 118 selecting another inputfield, such as the button 304. The data array 330 can be transmitted tothe content provider 106 responsive to a HTTP POST or GET method. Thedata processing system 102 can continue to interact with the digitalcomponent to complete the action identified in the input audio signal.For example, in the example illustrated in FIG. 3, the data processingsystem 102 can repeat ACTs of method 200 to select the shirt, checkoutor purchase the shirt, and then send a confirmation to the clientcomputing device 102.

The data processing system 102 can establish a communication session 322between the data processing system 102 and the computing device 104. Thecommunication session 322 can be established by a conversationalapplication programming interface. The communication session 322 can bea real-time, back-and-forth voice or audio based conversation session.The data processing system 102 can establish the communication session322 with the computing device 104 to retrieve additional information forthe action data structure or the data array. For example, the dataprocessing system 102 can transmit an output audio signal 326 withinstructions that cause the transducer of the computing device 104 togenerate a sound wave of “What is your preferred color.” The user canprovide a second input audio signal 328 in response to the output audiosignal 326. The natural language processor component 112 can process thesecond input audio signal 328 to identify the user's response, which inthis example can be “red.” The navigation component 118 can generate asecond data array 332 based on the interaction model and the responseidentified in the second input audio signal 328. The second data array332 can be transmitted to the content provider 106.

The data processing system 102 can establish a communication sessionwith a second computing device 104 that is associated with the user ofthe first computing device 104 that originally transmitted the inputaudio signal. For example, the first computing device 104 can be avoice-based digital assistant speaker system and the second computingdevice 104 can be the user's smart phone. The data processing system 102can request additional information or confirmation from the user via thesecond computing device 104. For example, in the example illustrated inFIG. 3, the data processing system 102 can provide two images ofselected shirts to the user's smart phone and request that the userselect one of the two shirts. Prior to completing a purchase or making areservation the data processing system 102 can request a verbalconfirmation via the first computing device 104 or a confirmation viathe second computing device 104 (e.g., the selection of a “buy” button).

FIG. 4 is a block diagram of an example computer system 400. Thecomputer system or computing device 400 can include or be used toimplement the system 100, or its components such as the data processingsystem 102. The data processing system 102 can include an intelligentpersonal assistant or voice-based digital assistant. The computingsystem 400 includes a bus 405 or other communication component forcommunicating information and a processor 410 or processing circuitcoupled to the bus 405 for processing information. The computing system400 can also include one or more processors 410 or processing circuitscoupled to the bus for processing information. The computing system 400also includes main memory 415, such as a random access memory (RAM) orother dynamic storage device, coupled to the bus 405 for storinginformation, and instructions to be executed by the processor 410. Themain memory 415 can be or include the data repository 124. The mainmemory 415 can also be used for storing position information, temporaryvariables, or other intermediate information during execution ofinstructions by the processor 410. The computing system 400 may furtherinclude a read only memory (ROM) 420 or other static storage devicecoupled to the bus 405 for storing static information and instructionsfor the processor 410. A storage device 425, such as a solid statedevice, magnetic disk or optical disk, can be coupled to the bus 405 topersistently store information and instructions. The storage device 425can include or be part of the data repository 124.

The computing system 400 may be coupled via the bus 405 to a display435, such as a liquid crystal display, or active matrix display, fordisplaying information to a user. An input device 430, such as akeyboard including alphanumeric and other keys, may be coupled to thebus 405 for communicating information and command selections to theprocessor 410. The input device 430 can include a touch screen display435. The input device 430 can also include a cursor control, such as amouse, a trackball, or cursor direction keys, for communicatingdirection information and command selections to the processor 410 andfor controlling cursor movement on the display 435. The display 435 canbe part of the data processing system 102, the client computing device104 or other component of FIG. 1, for example.

The processes, systems and methods described herein can be implementedby the computing system 400 in response to the processor 410 executingan arrangement of instructions contained in main memory 415. Suchinstructions can be read into main memory 415 from anothercomputer-readable medium, such as the storage device 425. Execution ofthe arrangement of instructions contained in main memory 415 causes thecomputing system 400 to perform the illustrative processes describedherein. One or more processors in a multi-processing arrangement mayalso be employed to execute the instructions contained in main memory415. Hard-wired circuitry can be used in place of or in combination withsoftware instructions together with the systems and methods describedherein. Systems and methods described herein are not limited to anyspecific combination of hardware circuitry and software.

Although an example computing system has been described in FIG. 4, thesubject matter including the operations described in this specificationcan be implemented in other types of digital electronic circuitry, or incomputer software, firmware, or hardware, including the structuresdisclosed in this specification and their structural equivalents, or incombinations of one or more of them.

For situations in which the systems discussed herein collect personalinformation about users, or may make use of personal information, theusers may be provided with an opportunity to control whether programs orfeatures that may collect personal information (e.g., information abouta user's social network, social actions or activities, a user'spreferences, or a user's location), or to control whether or how toreceive content from a content server or other data processing systemthat may be more relevant to the user. In addition, certain data may beanonymized in one or more ways before it is stored or used, so thatpersonally identifiable information is removed when generatingparameters. For example, a user's identity may be anonymized so that nopersonally identifiable information can be determined for the user, or auser's geographic location may be generalized where location informationis obtained (such as to a city, postal code, or state level), so that aparticular location of a user cannot be determined. Thus, the user mayhave control over how information is collected about him or her and usedby the content server.

The subject matter and the operations described in this specificationcan be implemented in digital electronic circuitry, or in computersoftware, firmware, or hardware, including the structures disclosed inthis specification and their structural equivalents, or in combinationsof one or more of them. The subject matter described in thisspecification can be implemented as one or more computer programs, e.g.,one or more circuits of computer program instructions, encoded on one ormore computer storage media for execution by, or to control theoperation of, data processing apparatuses. Alternatively or in addition,the program instructions can be encoded on an artificially generatedpropagated signal, e.g., a machine-generated electrical, optical, orelectromagnetic signal that is generated to encode information fortransmission to suitable receiver apparatus for execution by a dataprocessing apparatus. A computer storage medium can be, or be includedin, a computer-readable storage device, a computer-readable storagesubstrate, a random or serial access memory array or device, or acombination of one or more of them. While a computer storage medium isnot a propagated signal, a computer storage medium can be a source ordestination of computer program instructions encoded in an artificiallygenerated propagated signal. The computer storage medium can also be, orbe included in, one or more separate components or media (e.g., multipleCDs, disks, or other storage devices). The operations described in thisspecification can be implemented as operations performed by a dataprocessing apparatus on data stored on one or more computer-readablestorage devices or received from other sources.

The terms “data processing system,” “computing device 104,” “component,”or “data processing apparatus” encompass various apparatuses, devices,and machines for processing data, including by way of example aprogrammable processor, a computer, a system on a chip, or multipleones, or combinations of the foregoing. The apparatus can includespecial purpose logic circuitry, e.g., an FPGA (field programmable gatearray) or an ASIC (application specific integrated circuit). Theapparatus can also include, in addition to hardware, code that createsan execution environment for the computer program in question, e.g.,code that constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, a cross-platform runtimeenvironment, a virtual machine, or a combination of one or more of them.The apparatus and execution environment can realize various differentcomputing model infrastructures, such as web services, distributedcomputing and grid computing infrastructures. For example, the directaction API 116, the NLP component 112, and other data processing system102 components can include or share one or more data processingapparatuses, systems, computing devices, or processors.

A computer program (also known as a program, software, softwareapplication, app, script, or code) can be written in any form ofprogramming language, including compiled or interpreted languages,declarative or procedural languages, and can be deployed in any form,including as a stand-alone program or as a module, component,subroutine, object, or other unit suitable for use in a computingenvironment. A computer program can correspond to a file in a filesystem. A computer program can be stored in a portion of a file thatholds other programs or data (e.g., one or more scripts stored in amarkup language document), in a single file dedicated to the program inquestion, or in multiple coordinated files (e.g., files that store oneor more modules, sub programs, or portions of code). A computer programcan be deployed to be executed on one computer or on multiple computersthat are located at one site or distributed across multiple sites andinterconnected by a communication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs (e.g., components of the data processing system 102)to perform actions by operating on input data and generating output. Theprocesses and logic flows can also be performed by, and apparatuses canalso be implemented as, special purpose logic circuitry, e.g., an FPGA(field programmable gate array) or an ASIC (application specificintegrated circuit). Devices suitable for storing computer programinstructions and data include all forms of non-volatile memory, mediaand memory devices, including by way of example semiconductor memorydevices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks,e.g., internal hard disks or removable disks; magneto optical disks; andCD ROM and DVD-ROM disks. The processor and the memory can besupplemented by, or incorporated in, special purpose logic circuitry.

The subject matter described herein can be implemented in a computingsystem that includes a back end component, e.g., as a data server, orthat includes a middleware component, e.g., an application server, orthat includes a front end component, e.g., a client computer having agraphical user interface or a web browser through which a user caninteract with the computer system 100 or other elements describedherein, or a combination of one or more such back end, middleware, orfront end components. The components of the system can be interconnectedby any form or medium of digital data communication, e.g., acommunication network. Examples of communication networks include alocal area network (“LAN”) and a wide area network (“WAN”), aninter-network (e.g., the Internet), and peer-to-peer networks (e.g., adhoc peer-to-peer networks).

The computing system such as system 100 can include clients and servers.A client and server are generally remote from each other and typicallyinteract through a communication network (e.g., the network 105). Therelationship of client and server arises by virtue of computer programsrunning on the respective computers and having a client-serverrelationship to each other. A server can transmit data (e.g., datapackets representing a content item) to a computing device 104 (e.g.,for purposes of displaying data to and receiving user input from a userinteracting with the computing device 104). Data generated at thecomputing device 104 (e.g., a result of the user interaction) can bereceived from the computing device 104 at the server (e.g., received bythe data processing system 102 from the computing device 104 or contentprovider 106).

While operations are depicted in the drawings in a particular order,such operations are not required to be performed in the particular ordershown or in sequential order, and all illustrated operations are notrequired to be performed. Actions described herein can be performed in adifferent order.

The separation of various system components does not require separationin all implementations, and the described program components can beincluded in a single hardware or software product. For example, the NLPcomponent 112 can be a single component, app, or program, or a logicdevice having one or more processing circuits, or part of one or moreservers of the data processing system 102.

Having now described some illustrative implementations, it is apparentthat the foregoing is illustrative and not limiting, having beenpresented by way of example. In particular, although many of theexamples presented herein involve specific combinations of method actsor system elements, those acts and those elements may be combined inother ways to accomplish the same objectives. Acts, elements andfeatures discussed in connection with one implementation are notintended to be excluded from a similar role in other implementations orimplementations.

The phraseology and terminology used herein is for the purpose ofdescription and should not be regarded as limiting. The use of“including” “comprising” “having” “containing” “involving”“characterized by” “characterized in that” and variations thereofherein, is meant to encompass the items listed thereafter, equivalentsthereof, and additional items, as well as alternate implementationsconsisting of the items listed thereafter exclusively. In oneimplementation, the systems and methods described herein consist of one,each combination of more than one, or all of the described elements,acts, or components.

Any references to implementations or elements or acts of the systems andmethods herein referred to in the singular may also embraceimplementations including a plurality of these elements, and anyreferences in plural to any implementation or element or act herein mayalso embrace implementations including only a single element. Referencesin the singular or plural form are not intended to limit the presentlydisclosed systems or methods, their components, acts, or elements tosingle or plural configurations. References to any act or element beingbased on any information, act or element may include implementationswhere the act or element is based at least in part on any information,act, or element.

Any implementation disclosed herein may be combined with any otherimplementation or embodiment, and references to “an implementation,”“some implementations,” “one implementation” or the like are notnecessarily mutually exclusive and are intended to indicate that aparticular feature, structure, or characteristic described in connectionwith the implementation may be included in at least one implementationor embodiment. Such terms as used herein are not necessarily allreferring to the same implementation. Any implementation may be combinedwith any other implementation, inclusively or exclusively, in any mannerconsistent with the aspects and implementations disclosed herein.

References to “or” may be construed as inclusive so that any termsdescribed using “or” may indicate any of a single, more than one, andall of the described terms. For example, a reference to “at least one of‘A’ and ‘B’” can include only ‘A’, only ‘B’, as well as both ‘A’ and‘B’. Such references used in conjunction with “comprising” or other openterminology can include additional items.

Where technical features in the drawings, detailed description or anyclaim are followed by reference signs, the reference signs have beenincluded to increase the intelligibility of the drawings, detaileddescription, and claims. Accordingly, neither the reference signs northeir absence have any limiting effect on the scope of any claimelements.

The systems and methods described herein may be embodied in otherspecific forms without departing from the characteristics thereof. Theforegoing implementations are illustrative rather than limiting of thedescribed systems and methods. Scope of the systems and methodsdescribed herein is thus indicated by the appended claims, rather thanthe foregoing description, and changes that come within the meaning andrange of equivalency of the claims are embraced therein.

What is claimed:
 1. A system to retrieve digital components in a voiceactivated data packet based computer network environment, comprising: anatural language processor component executed by a data processingsystem to receive, via an interface of the data processing system, datapackets comprising an input audio signal detected by a sensor of acomputing device; the natural language processor component executed bythe data processing system to parse the input audio signal to identify arequest, a content provider, and a trigger keyword corresponding to therequest; a direct action application programming interface executed bythe data processing system to generate, based on the trigger keyword, anaction data structure responsive to the request and the contentprovider; a navigation component executed by the data processing systemto: establish, responsive to the identification of the content providerfrom the input audio signal, a session with the content provider;receive a digital component, via the session from the content providerusing the request identified from the input audio signal, the digitalcomponent comprising one or more input elements for a graphical userinterface, the digital component unable to accommodate voice-basedinteraction with any of the one or more input elements; headlesslyrender an image corresponding to the digital component comprising theone or more input elements received via the session without thegraphical user interface for the digital component; select aninteraction model associated with the content provider to identify theone or more input elements in the rendered image corresponding to thedigital component received from the content provider; generate, usingthe interaction model for the one or more input elements of the digitalcomponent, a data array to include the action data structure; providethe data array as an input to at least one of the one or more inputelements of the rendered image corresponding to the digital component;and the data processing system to transmit, via the interface, the dataarray to the content provider to fulfill the request identified from theinput audio signal.
 2. The system of claim 1, comprising the dataprocessing system to: select, from a database stored in memory, atemplate based on the trigger keyword and the content provider; andgenerate the action data structure based on the template.
 3. The systemof claim 1, comprising: a conversational application programminginterface to receive, in a communication session between the dataprocessing system and the computing device established via theinterface, data packets comprising a second input audio signal detectedby the sensor of the computing device; the natural language processorcomponent to parse the second input audio signal to identify a response;and the navigation component to generate a second data array based onthe interaction model and the response.
 4. The system of claim 1,comprising: the data processing system configured to provide the dataarray to an input element of the rendered digital component.
 5. Thesystem of claim 1, comprising the data processing system to: determine anumber of previous sessions with the content provider; and select theinteraction model based on the number of previous sessions with thecontent provider.
 6. The system of claim 5, comprising the dataprocessing system to: select a generic interaction model responsive tothe number of previous sessions being less than a predeterminedthreshold.
 7. The system of claim 1, comprising the data processingsystem to: identify a characteristic of an image in the rendered digitalcomponent.
 8. The system of claim 1, comprising the navigation componentto: generate the interaction model based on a previous session with thecontent provider.
 9. The system of claim 1, comprising a conversationalapplication programming interface to: establish a communication sessionbetween the data processing system and the computing device; andtransmit, via the interface, data packets comprising an output audiosignal to the computing device.
 10. The system of claim 1, wherein thedigital component is a webpage.
 11. A method to retrieve digitalcomponents in a voice activated data packet based computer networkenvironment, comprising: receiving, by a natural language processorcomponent executed by a data processing system and via an interface ofthe data processing system, data packets comprising an input audiosignal detected by a sensor of a computing device; parsing, by thenatural language processor component, the input audio signal to identifya request, a content provider, and a trigger keyword corresponding tothe request; generating, by a direct action application programminginterface, based on the trigger keyword, an action data structureresponsive to the request and the content provider; establishing, by anavigation component, responsive to the identification of the contentprovider from the input audio signal, a session with the contentprovider; receiving, by the navigation component, via the session fromthe content provider using the request identified from the input audiosignal, a digital component comprising one or more input elements for agraphical user interface, the digital component unable to accommodatevoice-based interaction with any of the one or more input elements;headlessly rendering, by the navigation component, an imagecorresponding to the digital component comprising the one or more inputelements without the graphical user interface for the digital component;selecting, by the navigation component, an interaction model associatedwith the content provider to identify one or more input elements in therendered image corresponding to the digital component received from thecontent provider; generating, by the navigation component, using theinteraction model for the one or more input elements of the digitalcomponent, a data array to include the action data structure; providing,by the navigation component, the data array as an input to at least oneof the one or more input element of the rendered image corresponding tothe digital component; and transmitting, by the data processing systemand via the interface, the data array to the content provider to fulfillthe request identified from the input audio signal.
 12. The method ofclaim 11, comprising: selecting, from a database stored in memory, atemplate based on the trigger keyword and the content provider; andgenerating the action data structure based on the template.
 13. Themethod of claim 11, comprising: receiving, by a conversationalapplication programming interface and via a communication sessionbetween the data processing system and the computing device, datapackets comprising a second input audio signal detected by the sensor ofthe computing device; parsing, by the natural language processorcomponent, the second input audio signal to identify a response; andgenerating, by the navigation component, a second data array based onthe interaction model and the response.
 14. The method of claim 11,comprising: providing the data array to an input element of the digitalcomponent.
 15. The method of claim 11, comprising: determining a numberof previous sessions with the content provider; and selecting theinteraction model based on the number of previous sessions with thecontent provider.
 16. The method of claim 15, comprising: selecting ageneric interaction model responsive to the number of previous sessionsbeing less than a predetermined threshold.
 17. The method of claim 11,comprising: rendering the digital component without a graphical userinterface.
 18. The method of claim 11, comprising: identifying acharacteristic of an image in the digital component.
 19. The method ofclaim 11, comprising: generating the interaction model based on aprevious session with the content provider.